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Abstract 


We  review  previous  efforts  to  recover  surface  shape  from  image  irradiance  in  order  to 
assses  what  can  and  cannot  be  accomplished.  We  consider  the  informational  requirements 
and  restrictions  of  these  approaches.  In  dealing  with  the  question  of  what  surface  parameters 
can  be  recovered  locally  from  image  shading,  we  show  that,  at  most,  shading  determines 
relative  surface  curvature,  i.e,  the  ratio  of  surface  curvature  measured  in  orthogonal  image 
directions.  The  relationship  between  relative  surface  curvature  and  the  second  derivatives 
of  image  irradiance  is  independent  of  other  scene  parameters,  but  insufficient  to  determine 
surface  shape.  This  result  places  in  perspective  the  difficulty  encountered  in  previous 
attempts  to  recover  surface  orientation  from  image  shading. 


1.  Introduction 

The  determination  of  land  cover  from  aerial  imagery  is  a  task  that  photo  interpreters 
accomplish  by  using  both  the  image  data  and  their  knowledge  of  the  structure  of  the  world. 
The  image  data  encodes  the  complex  process  whereby  light  is  reflected  from  a  surface.  The 
surface  shape,  the  surface  albedo,  the  position  of  the  lighting  sources,  and  the  functional 
form  of  the  reflectance  properties  of  the  material  are  elements  of  this  encoding.  The  human 
visual  system  interprets  image  data  as  a  3-D  model  of  the  scene,  distinguishes  among  different 
surface  materials,  and  ascertains  the  position  of  the  lighting  sources.  It  is  difficult  to  believe 
that  a  machine  vision  system  can  achieve,  say,  surface  material  differentiation  without 
simultaneously  being  able  to  recover  the  surface  shape  and  the  other  parameters  that  are 
needed  to  explain  the  detected  image  intensity.  Of  course,  it  may  be  possible  to  use  special 
sensors  and  multiple  information  sources  to  make  it  unnecessary  to  reconstruct  a  complete 
3-D  mode!  of  the  scene,  but  it  would  be  surprising  if  such  specialization  could  retain  sufficient 
generality  to  be  useful  over  a  range  of  remote  sensing  tasks,  e.g.,  in  both  renewable  and 
nonrenewable  resources. 

The  machine  vision  approach  of  simultaneously  recovering  all  the  parameters  necessary 
to  account  for  image  intensity  is  expressed  in  the  notion  of  intrinsic  images  [l]  (or  the  2^- 
D  sketch  [2]).  These  intrinsic  images  can  be  thought  of  as  overlays,  each  specifying  the 
value  of  one  parameter  that  goes  into  the  formula  for  calculating  the  image  intensity.  The 

The  research  reported  herein  was  supported  by  the  Defense  Advanced  Research  Projects  Agency  under 
Contract  MDA903-83-C-0027  and  by  the  National  Aeronautics  and  Space  Administration  under  Contract 
NASA  9-16664.  These  contracts  are  monitored  by  the  U.S.  Army  Engineer  Topographic  Laboratory  and  by 
the  Texas  A&M  Research  Foundation  for  the  Lyndon  B.  Johnson  Space  Center. 
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images  are  not  independent;  if  one  is  to  be  varied,  the  others  must  be  also  —  so  that 
the  predicted  image  intensity  remains  invariant  (and  equal  to  the  observed  value).  The 
notional  division  of  an  image  into  particular  intrinsic  images  would  be  of  little  merit  unless 
one  believed  that  estimates  of  eaich  intrinsic  image  could  be  obtained  by  models  that  were 
largely  independent  of  the  other  intrinsic  images.  While  models  have  been  proposed  to 
recover  various  intrinsic  images,  there  have  been  considerable  efforts  made  to  recover  the 
scene’s  3-D  shape,*  in  particular  the  surface  orientation  at  each  image  point.  These  ‘shape- 
from-...’  models  embody  a  structure  that  would  allow  shape  to  be  recovered  principally 
from  a  single  measure,  e.g.,  texture,  contour,  or  shading.  While  ‘shape-from-...’  models  are 
not  seen  as  complete  solutions  to  shape  reconstruction,  there  is  an  implicit  expectation  in 
their  title  that  shape  estimates  can  be  calculated  from  their  respective  measures.  Here  we 
review  the  work  we  and  others  have  done  towards  the  goal  of  recovering  surface  shape  from 
image  shading.  Is  it  attainable  —  or  is  it  myth? 


The  importance  of  shape  recovery  is  clear;  if  the  shape  is  known,  surface  albedo,  and  the 
other  parameters  that  determine  image  intensity  are  obtainable.  Land  cover  differentation 
is  dependent  on  knowing  the  [relative]  surface  albedo,  rather  than  image  parameters,  such 
as  intensity.  If  we  cannot  recover  shape,  the  intrinsic  image  approach  offers  little  as  a  model 
for  perception.  Shading  is  only  one  source  of  shape  information.  Edge  information  is  of 
great  importance,  but  there  is  little  occlusion  in  aerial  images.  The  ability  to  recover  shape 
from  shading  seems  more  critical  in  the  case  of  aerial  imagery  than  for  most  other  types  of 
imagery. 


We  first  review  three  research  efforts:  those  of  Horn  and  his  colleagues  [3-8],  Pentland 
[9],  and  our  own  [lO],  —  to  determine  what  can  and  cannot  be  accomplished,  and  to  consider 
the  informational  requirements  and  limitations  of  these  approaches.  We  discuss  the  dilemma 
of  local  computation  versus  global  constraint  propagation  and  seek  to  ascertain  what  can 
be  computed  locally,  and  how  information  can  be  propagated  across  an  image.  Finally,  we 
seem  to  be  left  with  the  conclusion  that  shading,  when  viewed  as  a  single  source  of  shape 
information,  is  an  insufficient  source  for  the  recovery  of  surface  shape.  Shape  cannot  be 
obtained  from  shading  alone.  However,  we  are  able  to  characterize  the  scene  information 
that  shading  provides. 


An  alternative  approach  to  recovering  shape  from  shading  is  model  based.  Can  we 
determine  which  model,  from  a  set  of  models,  best  describes  the  image  data?  This  approach 
is  dependent  on  discovering  a  small  set  of  easily  distinguishable  models  that  adequately 
describe  the  surfaces  encountered.  Industrial  inspection,  rather  than  remote  sensing  of  the 
environment,  appears  better  suited  to  a  model  based  procedure.  In  this  assessment  we  do 
not  consider  this  related,  but  essentially  different  approach. 

^We  use  the  expression  surface  shape  to  denote  both  the  intrinsic  properties  of  the  surface,  e.g.,  cylindrical, 
and  the  orientation  of  the  surface  in  space.  Elsewhere,  shape  is  sometimes  used  to  denote  only  the  intrinsic 
properties  of  the  surface,  not  its  orientation  in  space. 
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2. 


Approaches  to  Shape  from  Shading 


2.1  Horn  and  Colleagues 

A  study  by  Horn  [3,4,11]  of  the  relationship  among  image  irradiance,"  surface  shape, 
surface  albedo,  and  illumination  conditions  led  to  formulation  of  the  image  irradiance 
equation,  which  states  that  image  irradiance  is  proportional  to  scene  radiance.^  This  is 
expressed  by  the  equation 

I  =  R  , 

where  I  is  the  image  irradiance  as  a  function  of  the  image  coordinates,  and  R  is  the  scene 
radiance  as  a  function  of  the  scene  parameters.  Of  course,  this  equation  relates  the  image 
irradiance  at  an  position  in  the  image  to  the  scene  radiance  at  its  corresponding  scene 
position.  Implicit  in  this  equation  is  an  assumption  of  orthographic  projection.  However, 
such  an  assumption,  to  avoid  complexity  in  the  mathematical  formulation,  is  a  minor 
restriction  and  does  not  detract  from  the  generality  of  the  model. 

Image  irradiance  is  a  function  of  the  image  coordinates  x  and  y,  but  scene  radiance 
is  a  function  of  the  illumination  strength,  its  position,  the  surface  albedo,  and  the  surface 
orientation.  For  the  formulations  reviewed  here,  we  find  that  a  number  of  assumptions  are 
made  so  that  scene  radiance  can  be  considered  a  function  of  the  surface  orientation  variables 
only;  constant  values  are  used  for  the  illumination  strength,  its  position,  and  for  the  surface 
albedo.  That  is,  shape-from-shading  is  analyzed  for  the  simplified  case  of  a  constant  light 
source  and  constant  surface  albedo.  The  restriction  to  a  constant  light  source  is  not  only  a 
good  approximation  of  the  situation  we  experience  daily  (and  an  excellent  approximation  for 
a  photograph),  but  also  corresponds  to  the  difficulty  confronting  the  human  visual  system 
when  this  constancy  is  not  met,  e.g.,  under  strohe  lighting.  The  assumption  of  constant 
albedo  is  harder  to  justify,  since  nature  obviously  exhibits  variable  albedo.  Still,  when  we 
consider  the  manner  in  which  facial  make-up  is  used  to  alter  the  perceived  shape  of  the 
face,  it  may  well  be  that  continuous  changes  in  albedo  are  processed  by  the  human  visual 
system  as  if  they  were  constant.  Notwithstanding  the  justification  for  constant  albedo,  it  is 
unlikely  that  shape-from-shading  can  be  solved  for  the  case  of  variable  albedo  if  it  cannot 
be  solved  for  constant  albedo.  Such  a  restriction  is  in  effect  a  case  analysis  to  determine  if 
shading  provides  sufficient  shape  information  in  a  less-than-general  model. 

In  the  formulations  under  review,  various  parameterizations  of  surface  orientation  have 
been  used.  The  two  we  specify  are  (i)  surface  gradients,  i.e.,  the  partial  derivatives  of  depth, 
z,  with  respect  to  the  scene  (and  image)  coordinates  x  and  y,  and  (ii)  components  of  the 
surface  normal,  i.e.,  I  and  m,  the  x  and  y  components  of  the  surface  normal.  Using  the 
notation,  p  —  and  7  =  we  note  the  equivalence  of  the  parameterizations 


-/ 


P  = 


,  and  q 


—m 


^Image  irradiance  is  the  light  flux  per  unit  area  falling  on  the  image,  i.e.,  incident  flux  density. 

®Scene  radiance  is  the  light  flux  per  unit  projected  area  per  unit  solid  angle  emitted  from  the  scene,  i.e., 
emitted  flux  density  per  unit  solid  angle. 
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The  image  irradiance  equation  is  usually  expressed  as 

/(x,  y)  =  i?{p,  g)  ,or  I{x,y)  =  R{l,Tn)  , 

and  we  shall  use  both  forms  to  express  the  relationship  between  image  irradiance  and  scene 
radiance  for  the  case  of  constant  illumination  and  constant  albedo.  Asp  =  and  g  = 
we  see  that  the  image  irradiance  equation  is  a  first-order  partial  differential  equation  and, 
if  I  and  i?  are  known,  we  could  (at  least  in  principle)  solve  the  differential  equation  and 
recover  the  depth,  z. 

To  have  an  explicit  form  for  R,  we  must  have  a  model  for  the  type  of  reflection  occurring 
at  the  scene  surfaces.  In  the  work  reviewed  here  the  surface  is  assumed  to  be  a  perfectly 
uniform  diffuse  reflector,  i.e.,  the  scene  radiance  is  isotropic.^  While  this  model  is  invalid 
as  a  description  of  specular  reflection,  scene  radiance  in  the  natural  world,  (except  for 
specific  situations,  such  as  water  surfaces),  may  be  approximated  by  such  a  description. 
The  expression  for  scene  radiance  in  this  case  is  [lOj 

R(l,  m)  =  al  +  hm  +  c\/l  —  P  —  rrfi 


or,  equivalently. 


R{p,  9) 


(— flp  —  6g  -1-  c) 

\/l  +  p2  +  g2 


where  a, A,  and  c  are  constants  expressing  illumination  strength,  its  position,  and  the  surface 
albedo. 

The  approach  taken  by  Horn  and  his  colleagues  [3-8]  is  to  solve  the  first-order  partial 
differential  equation. 


(— qp  —  Ag+  c) 

\/l  +  p2  +  g2 


assuming  that  a, A,  and  c  are  known  —  i.e.,  the  surface  albedo,  and  the  illumination  strength, 
and  its  position.  While  this  need  to  know  scene  parameters  may  seem  over-restrictive,  such 
information  may  come  from  other  components  of  a  vision  system.  The  need  to  know  the 
illumination  position  does  not  seem  to  be  a  major  drawback  of  this  approach,  but  the 
requirement  that  the  scene  albedo  be  known  is  troublesome.  If  the  conceptual  model  of 
intrinsic  images  is  to  be  followed,  the  inability  to  decouple  surface  orientation  from  surface 
albedo  would  seem  fundamental.  Regardless  of  this  difficulty,  the  question  of  whether  shape 
can  be  recovered  in  a  limited  domain  is  basic  to  the  investigation  of  vision. 

Two  approaches  to  solving  the  image  irradiance  equation  are  direct  integration  [3,4], 
and  iterative/relaxation  techniques  [5-8].  The  direct  integration  approach  has  been  carried 
out  generally  in  those  circumstances  in  which  I{x,  y)  and  its  derivatives  can  be  determined 
for  all  X  and  y,  i.e.,  for  a  spatially  unquantized,  continuous-tone  image.  The  method  used 
is  the  standard  technique  of  characteristic  strips  for  solving  a  first-order  hyperbolic  partial 


■'This  situation  is  also  called  Lambertian  reflectance,  after  Lambert,  who  proposed  a  point  reflection  model 
(in  which  the  reflected  flux  per  unit  surface  area  per  unit  solid  angle  varied  as  the  cosine  of  the  angle 
between  the  surface  normal  and  the  viewing  direction)  to  account  for  the  observation  that  matt  surfaces 
looked  equally  bright  from  any  viewing  position. 
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differential  equation  [3,4].  Starting  with  a  point  at  which  the  surface  orientation  is  known, 
integration  moves  along  a  curve  in  the  image.  This  curve  is  dictated  hy  the  image.  Adjacent 
curves  generally  are  not  ‘parallel’,  which  makes  it  difficult  to  get  complete  coverage  of  the 
image.  Interpolation  between  these  curves  —  or  strips,  as  they  are  usually  called  —  to  find 
initial  values  to  commence  an  intervening  strip  integration,  involves  complex  procedures. 
As  far  as  digital  images  are  concerned,  direct  integration  would  he  hard  to  organize,  even 
if  we  were  first  to  model  the  intensities  to  obtain  a  continuous  form  for  I{x,  y). 

As  is  the  case  with  most  partial  differential  equations,  it  should  be  noted  that  the  image 
irradiance  equation  has  many  solutions  [12].  The  boundary  conditions  (in  the  above  method 
the  initial  values  for  a  strip)  are  vital  in  selecting  the  solution  that  describes  the  surface  in 
the  image.  Should  the  image  irradiance  equation  be  ‘underconstrained’  in  the  sense  that,  for 
a  given  I{x,  y),  it  admits  solutions  that  encompass  a  wide  range  of  surface  types  with  similar 
boundary  values,  we  might  then  expect  numerical  error  to  defeat  attempts  at  numerical 
integration.  In  such  cases,  errors  ‘mix  in’  other  solutions  that  can  eventually  dominate  the 
recovered  solution,  even  though  they  may  be  excluded  by  the  boundary  conditions.  The 
method  of  direct  integration  has  been  demonstrated  on  simple  images  [3].  These  examples 
required  only  a  small  number  of  integration  steps.  Numerical  instability  has  also  been 
reported  [12]. 

The  other  approach  used  to  solve  the  image  irradiance  equation  is  relaxation. 
Relaxation  procedures  avoid  numerical  instability,  but  face  the  problem  of  convergence. 
However,  they  do  have  the  advantage  of  being  directly  applicable  to  digital  images,  i.e., 
spatially  quantized,  discrete-tone  Images.  The  relaxation  (or  iterative)  approach  views  the 
image  irradiance  equation  not  as  a  differential  equation,  but  as  an  algebraic  constraint.  Por 
pixel  (i,  j), 

where  /,-,j  is  the  image  irradiance  for  the  (i,  y)th  pixel,  and  p,-_y,and  specify  the  surface 
orientation  of  the  surface  patch  that  is  imaged  at  pixel  (i,  j).  As  an  algebraic  constraint,  the 
image  irradiance  equation  relates  image  irradiance  to  the  two  surface  orientation  variables, 
p,‘,y,and  qij.  In  viewing  the  image  irradiance  equation  as  a  algebraic  constraint,  we  lose 
the  interrelationship  of  p,-j,  q{j,  and  their  neighboring  values,  a  relationship  inherent  in 
the  differential  equation.  To  compensate  for  this  loss,  an  additional  constraint  must  be 
introduced  that  relates  p,-,y,and  to  their  neighboring  values.  Such  a  relationship  is 
essential  for  a  relaxation  procedure.  The  relationship  usually  introduced  attempts  to  capture 
the  notion  of  surface  smoothness  [7,8,  10,13].  The  particular  form  of  the  smoothness 
constraint  may,  for  example,  require  that  p;^y,and  he  equal  to  the  mean  values  of 
neighboring  p’s  and  q^s.  For  any  trial  values  for  pfj-,and  qij,  the  constraint  imposed  by 
the  image  irradiance  equation  and  the  constraint  resulting  from  surface  smoothness  will  not 
be  completely  satisfied.  The  residual  equation  formed  from  each  constraint  specifies  how 
well  that  constraint  is  satisfied.  If  is  the  sum  of  the  [absolute  values  of  the]  residuals 
from  both  the  image  irradiance  constraint  and  the  surface  smoothness  constraint  for  the 
(i,  y)th  pixel,  then,  for  trial  values  of  p  and  q  for  every  image  pixel,  the  total  residual  error 
is 

i,j€image 
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The  allocation  of  surface  orientations  to  all  pixels  should  minimize  this  total  error  —  that 


is, 


dpij 

IL 


=  0 
=  0 


V  i,  j  6  image  , 

V  j ,  j  6  image  . 


From  these  equations  we  obtain  an  iterative  scheme  for  updating  the  values  of  p  and  q  so 
that  they  are  compatible  with  their  neighboring  values,  as  well  as  with  the  image  irradiance 
equation  [7,8,10].  If  such  a  scheme  is  convergent,  we  have  a  procedure  for  obtaining  shape 
from  shading. 

It  should  be  noted  that  the  relaxation  schemes,  that  use  the  foregoing  approach  are 
possible  only  because  the  smoothness  constraint  relates  the  values  at  one  pixel  to  those  of  its 
neighboring  pixels.  The  boundary  conditions  needed  for  selecting  a  particular  solution  from 
the  solution  set  of  the  iterative  scheme  are  propagated  by  the  smoothness  constraint,  not 
the  image  irradiance  equation.  Compared  with  the  direct-integration  approach,  information 
propagation  in  the  relaxation  scheme  uses  a  different  mechanism.  We  must  remember  this 
when  we  assess  results. 

Success  with  these  methods  has  generally  been  limited  to  small  images,  (usually  fewer 
than  30  x  30  pixels),  of  nearly  spherical  or  saddle  surfaces  [7,8,10,14].  For  an  effective 
relaxation  scheme,  the  initial  solution  should  have  no  effect  on  the  surface  recovered.  This 
unfortunately  is  not  the  case  [10].  Boundary  conditions  are  not  propagated  more  than 
a  few  pixels  by  the  smoothness  constraints  [7,10].  Surface  recovery  from  large  images, 
(bigger  than  30  x  30  pixels),  is  ineffectual  for  this  reason.  As  a  consequence  of  the  fact 
that  smoothness  is  used  as  information  propagator,  assumptions  (albeit  weak  ones)  have 
been  made  about  surface  shape.  Shading  as  a  constraint,  and  smoothness  as  a  surface  type, 
appears  insufficient  to  provide  a  basis  for  an  effective  shape-from-shading  algorithm. 


2.2  Pentland 

The  approaches  to  solving  the  shape-from-shading  task  that  we  have  discussed  so  far 
have  all  been  based  on  constraint  propagation.  Direct  integration  is  a  spatially  serial  solution 
to  the  propagation  problem,  while  relaxation  attempts  to  achieve  this  propagation  with  a 
temporally  serial  solution;  in  other  words,  relaxation  employs  local  processing,  hut  it  must 
iterate  until  enough  cycles  have  passed  to  allow  information  to  propagate  spatially.  Purely 
local  computation  of  scene  parameters,  on  the  other  hand,  is  not  a  propagation  method. 
While  this  kind  of  computation  can  use  neighboring  data  —  and  not  just  of  the  nearest 
neighbors  —  it  must  provide  an  instant  solution.  It  cannot  iterate  and  therefore  it  does 
not  provide  a  temporally  serial  solution.  Such  an  approach  to  scene  parameter  computation 
avoids  the  numerical  instability  of  direct  integration  methods,  as  well  as  the  convergence  and 
propagation  problems  of  relaxation,  but  it  cannot  use  spatially  distant  scene  information. 
A  local  computation  can  use  global  information,  such  as  the  position  of  the  light  source, 
but  it  cannot  use  scene  details,  such  as  the  position  of  a  distant  edge.  Of  course,  the  reason 
for  carrying  out  purely  local  computation  stems  from  the  hypothesis  that  such  scene  detail 
is  not  involved  in  the  computation  at  this  level  in  the  visual  system.  Can  shading  provide 
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sufficient  local  information  to  allow  recovery  of  surface  shape  by  purely  local  computation? 
This  is  the  question  addressed  by  Pentland  [9]. 

The  inadequacy  of  local  image  measurements  for  specifying  surface  orientation  can  he 
understood  by  counting  the  variables  needed  to  specify  various  image  measurements.  Let 
us  consider  the  case  of  a  uniformly  diffuse  reflecting  surface.  Image  irradiance  (I  measure¬ 
ment)  is  a  function  of  surface  orientation  (2  parameters),  the  product  of  surface  albedo  and 
illumination  strength  (1  parameter),  and  the  position  of  the  light  source  (2  parameters).  The 
gradients  of  image  irradiance  (2  measurements)  are  functions  of  the  same  variables  as  image 
irradiance  and,  additionally,  are  functions  of  surface  curvature  (3  parameters).  The  second 
derivatives  of  image  irradiance  (3  measurements)  are  functions  of  all  the  variables  men¬ 
tioned  above,  plus  the  rates  of  change  of  curvature  (4  parameters).  Because  higher  image- 
irradiance  derivatives  introduce  more  surface  shape  derivatives,  we  have  more  parameters 
than  measurements.  It  should  be  noted  that  a  knowledge  of  global  quantities,  such  as  the 
illumination  position  and  the  product  of  surface  albedo  and  illumination  strength,  is  not 
sufficient  to  allow  the  surface  orientation  to  be  computed  locally.  If  we  make  assumptions 
about  the  relationship  among  some  of  the  above  parameters,  we  can  produce  a  system  of 
equations  from  which  surface  orientation  can  be  calculated. 

Pentland  investigates  the  case  in  which  an  image  patch  of  a  uniformly  diffuse  reflecting 
surface  can  be  considered  identical  to  a  point  on  an  illuminated  sphere  whose  reflection  is 
also  uniformly  diffuse  [9].  He  calculates  the  orientation  of  the  surface  patch  on  the  sphere 
that  has  the  same  appearance  as  the  surface  patch  in  the  image.  Not  all  image  patches  can 
be  represented  by  points  on  an  illuminated  sphere.  Spheres  whose  reflection  is  uniformly 
diffuse  have  the  property 

^>0  , 

lyy 

where  subscripts  denote  partial  differentation  with  respect  to  those  subscripts.  There  are 
surfaces,  e.g.,  a  sinusoidal  surface,  for  which  ^  can  be  negative.  The  procedure  for  estimat- 
ing  surface  orientation  that  is  based  on  the  assumption  that  surfaces  can  be  approximated 
by  locally  spherical  patches  is  applicable  only  to  parts  of  an  image.  Notwithstanding  these 
restrictions,  an  important  aspect  of  the  assumption  of  local  sphericity  is  that  the  surface 
orientation  is  calculated  by  using  the  second  derivatives  of  image  irradiance  only,  i.e., 

1  — 

Im 

Im 

These  equations  are  derived  by  differentiating  the  image  irradiance  equation  and  noting 
that,  for  a  sphere,  f*  =  7,  fy  =  0,  m*  =  0,  and  my  =  where  r  is  the  sphere’s  radius. 

In  this  model,  surface  orientation  is  directly  dependent  on  neither  image  irradiance 
nor  on  the  first  derivatives  of  image  irradiance.  It  may  be  estimated  even  in  images 
that  exhibit  linear  changes  in  irradiance  induced  by  artifacts,  and  in  images  that  exhibit 
constant  illumination  levels  induced  by  atmospheric  effects,  such  as  backscatter.  More 
importantly,  the  formulas  are  independent  of  the  illumination  parameters  and  the  surface 
albedo.  In  exchange  for  acceptance  of  a  restrictive  assumption  with  respect  to  surface  type, 
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one  acquires  not  only  a  means  of  calculating  surface  orientation,  but  a  procedure  that  needs 
no  information  other  than  image  measurements  —  a  procedure,  in  effect,  that  is  matched 
to  the  notion  of  intrinsic  images. 

Even  in  those  areas  of  an  image  to  which  this  approximation  can  be  applied,  the 
assumption  that  a  surface  can  be  approximated  by  a  patch  with  the  same  curvature  in 
any  direction  needs  experimental  verification.  The  world  is  obviously  not  composed  of  such 
surfaces,  but  it  is  the  difference  between  the  estimated  and  the  actual  surface  orientation 
that  is  more  important  than  the  error  made  in  approximating  the  surface  by  a  spherical 
patch.  Application  of  the  above  formula  yields  qualitative  agreement  between  the  estimated 
and  actual  shape  in  synthetic  images  and  in  natural  images  of  simple  objects  [9],  (for  which 
^  is  generally  positive).  Shape  estimates  from  synthetic  images  of  ellipsoidal  surfaces 
are  ‘flatter’  than  the  actual  shapes.  It  should  be  noted  that  shape  estimates,  which  are 
integrated  surface  orientations,  often  appear  ‘better’  than  what  might  be  expected  on  the 
basis  of  the  surface  orientation  error.  An  algorithm  based  on  approximating  a  surface  patch 
by  a  spherical  one  seems  better  suited  for  computing  the  qualitative  shape  of  a  surface 
than  the  orientation  of  surface  elements.  Such  an  algorithm  is  applicable  only  to  thoses 
image  patches  that  are  consistent  with  the  interpretation  of  such  patches  as  points  on  a 
sphere.  The  conditions  necessary  for  enabling  this  kind  of  interpretation  have  not  been  fully 
characterized.  Alternative  models,  that  are  applicable  when  an  image  patch  is  inconsistent 
with  an  interpretation  that  it  is  a  point  on  a  sphere,  are  currently  unknown. 

In  principle,  because  image  irradiance  is  not  differentiable  at  boundaries,  we  cannot 
apply  the  above  method  there.  However,  unlike  propagation  methods  require  our  knowing 
boundary  positions  in  order  to  stop  computation,  the  local-computation  approach  may 
accomplish  this  simply  by  indicating  (through  its  failure  at  a  boundary)  where  the  boundary 
is. 

Pentland’s  approach  hinges  on  the  local-sphericity  assumption.  In  restricted  cir¬ 
cumstances  he  is  able  to  estimate  surface  orientation  directly  from  the  second  derivatives  of 
the  image  irradiance.  What  other,  perhaps  less  specific,  assumptions  can  be  made  that  allow 
shape  to  be  estimated  locally?  Before  attempting  to  answer  this,  we  review  the  shape-from- 
shading  formulation  we  have  previously  proposed  [10,15],  —  first,  to  assess  its  performance, 
then  to  provide  the  requisite  analytical  tools  for  answering  questions  about  local  computa¬ 
tion. 


2.3  Smith 

The  approach  taken  by  Horn  and  his  colleagues  provides  a  formulation  of  the  shape- 
from-shading  task  that  requires  knowledge  of  scene  parameters,  but  places  no  restriction 
on  the  surface  shape.  Calculation  of  surface  orientation  is  not  a  local  process,  and,  if 
surface  orientation  is  to  be  recovered,  knowledge  of  boundary  conditions  is  necessary. 
Pentland,  on  the  other  hand,  restricts  the  surface  shape  but  requires  no  scene  parameters,  no 
boundary  conditions,  and  derives  surface  orientation  by  purely  local  computation.  Is  there 
an  intermediate  position?  Is  there  a  formulation  that  neither  restricts  the  surface  shape  nor 
requires  knowledge  of  scene  parameters?  Of  course,  local  computation  seems  desirable  — 
but  is  it  worth  the  concomitant  cost  of  surface  type  restriction  or  the  requirement  that  scene 
parameters  be  known  a  priori?  The  formulation  previously  described  by  us,  takes  such  an 
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intermediate  position. 

For  a  uniformly  diffuse  reflecting  surface,  surface  orientation  is  related  to  image  ir- 
radiance  by  the  second-order  partial  differential  equations  [10] 

aOlxx  +  ^6mxx  -  a^lxy  -  Pirn^y  =  -  Xlhy  , 

”1"  O^^xy  ^^n^xy  X^^yy  X^^xy  j 


where 

Q  -  Tx^T^y  “  Jyft^x  ) 

^  —  ly^x  ^X^y  ) 

7  = /x^(l  —  -h  mx“(l  — /^)  +  2/xmx/m  , 

5  = /y®(l  —  m^) -I- my“(l  — -1- 2/yrny/T7i  , 

6  =  f*fy(l  —  m^)  +  mxmy{l  —  l^)  -I-  {Ixiriy  -h  lymx)lm  , 

X  —  IxTny  -  lyTUx  . 

These  equations  are  derived  from  the  image  irradiance  equation.  The  assumption  of 
uniformly  diffuse  reflection  relates  some  of  the  scene  parameters,  thereby  allowing  elimina¬ 
tion  of  parameters  that  specify  surface  albedo  and  illumination  conditions. 

The  assumption  that  surface  reflection  is  uniformly  diffuse  is  an  assumption  about 
the  physics  of  image  formation.  While  it  does  not  describe  the  reflectance  properties  of 
all  surface,  it  is  a  reasonable  approximation  to  most  surfaces  that  are  encountered  in  the 
natural  world.  For  any  formulation  of  the  relationship  between  shading  and  shape,  some 
assumptions  are  necessary.  Those  describing  properties  found  in  nature  are  more  palatable 
than  restrictions  for  which  little  a  priori  evidence  is  available. 

A  desirable  aspect  of  this  formulation  is  that  surface  orientation  is  not  related  to  image 
irradiance,  but  only  to  its  derivatives.  The  existance  of  constant  illumination  levels,  from 
atmospheric  scattering  or  fogging  of  photographic  images,  does  not  impede  the  potential 
for  shape  recovery.  Linear  changes  in  intensity,  however,  must  affect  the  shape  of  any 
recovered  surface.  A  more  important  aspect  of  this  formulation  is  its  independence  of  surface 
albedo.  Again  we  reiterate  that,  if  the  notion  of  intrinsic  images  is  to  be  useful  we  must 
find  models  that  decouple  surface  shape  from  surface  reflectance.  The  fact  that  knowledge 
of  the  illumination  conditions  is  not  required,  is  certainly  an  important  aspect,  but  less  so 
than  the  formulation’s  independence  of  surface  albedo. 

The  penalty  for  not  making  assumptions  about  surface  type  and  for  not  presupposing 
any  knowledge  of  scene  parameters,  such  as  illumination  conditions  and  surface  albedo,  is 
the  introduction  of  higher-order  derivatives  of  stirface  orientation  in  the  formulation,  as 
well  as  the  inability  to  calculate  surface  orientation  by  purely  local  computation.  Boundary 
conditions  are  necessary.  To  formulate  a  model  that  relates  surface  orientation  to  image 
irradiance  is  one  thing;  to  solve  it  for  that  orientation  is  another. 

The  second-order  partial  differential  equations  (given  above)  relating  surface  orientation 
and  image  irradiance  are  satisfied  by  solutions  to  the  first-order  partial  differential  equation 
X  =  0.  This  is  undesirable,  as  solutions  of  x  —  0  satisfy  the  surface-orientation-image- 
irradiance  equations  independently  of  the  image  measurements,  IxJyJxx,  ^yy,  and  Ixy.  The 
equation  x  =  0  characterizes  the  developable  surfaces,  e.g.,  a  cylinder  or  cone  (see  Appendix 
B);  derivation  of  the  above  surface-orientation-image-irradiance  equations  is  impossible  when 
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the  surface  is  developable,  i.e.,  singularly  curved.  The  surface-orientation-image-irradiance 
equations  are  appropiate  only  when  the  surface  is  doubly  curved.  For  singularly  curved 
surfaces,  the  appropiate  equations  relating  surface  orientation  and  image  irradiance  are 

~  +  rrixlm)  , 

I^TTly  JylTix  -  0  y 

0 

(These  equations  are  derived  independently  of  any  reflection  function,  i.e.,  they  apply  to  all 
types  of  reflection,  not  just  uniformly  diffuse  reflection.  See  Appendix  C.) 

If  the  surface-orientation-image-irradiance  equations  were  solved  by  analj'tic  proce¬ 
dures,  the  problems  posed  by  the  X  =  0  solutions  would  vanish,  as  such  solutions  would 
be  ruled  out  by  boundary  conditions.  However,  the  presence  of  such  solutions  heralds 
difficulties  for  numerical  methods,  as  the  inevitable  numerical  errors  will  mix  these  solutions 
into  the  recovered  surface  orientations.  Two  approaches  to  solving  the  surface-orientation- 
image-irradiance  equations  have  been  reported  [15].  These  approaches  are  direct  integration, 
which  is  implemented  by  finite-difference  formulas,  and  relaxation.  Both  require  additional 
information  in  the  form  of  boundary  conditions.  Both  fail  to  recover  surface  orientation. 
Direct  integration  correctly  recovers  the  surface  orientation  in  the  vincinity  of  the  boundary 
conditions,  but  is  ineffective  elsewhere.  The  reasons  for  failure  of  each  method  are  of  inter¬ 
est;  direct  integration  fails  because  numerical  instability,  makes  the  spatially  serial  method 
of  solution  impractical;  relaxation  fails  because  nonconvergence  makes  the  temporally  serial 
method  of  solution  infeasible.  These  direct  reasons  for  failure  mask  a  deeper  problem. 
The  model  is  ‘underconstrained’  from  the  standpoint  that  the  equations  are  insensitive  to 
surface  orientation.  They  are  more  sensitive  to  other  surface  parameters,  such  as  surface 
curvature  [15].  Underconstraint  of  the  model  can  account  for  lack  of  convergence  of  relaxa¬ 
tion  methods,  but  the  numerical  problems  in  direct  integration  highlight  the  difficulty  of 
spatial  information  propagation  by  a  mechanism  that  is  under  the  control  of  higher-order 
derivatives. 

The  surface-orientation-image-irradiance  equations  alone  do  not  form  the  basis  for  an 
algorithm  to  recover  surface  orientation;  they  do  provide  a  tool,  however,  for  examining  the 
constraint  shading  imposes  on  shape.  We  shall  subsequently  use  them  for  that  purpose. 


3.  Local  Computation  Versus  Global-Constraint  Propagation 

What  can  we  learn  from  these  various  approaches  to  shape-from-shading?  Direct 
integration  of  a  differential  model  is  an  inadequate  computational  tool.  Horn  and  his 
colleagues,  using  a  low-order  partial  differential  equation,  show  that  some  propagation  of 
information  is  possible  —  but  numerical  instability  poses  a  difficulty  even  for  a  first-order 
equation.  This  limited  success  with  direct  integration  is  unlikely  to  be  upgradable  to  a 
solution  procedure  for  natural  scenes.  Since  higher-order  formulations  are  plagued  with 
numerical  instability  they  do  not  offer  any  prospect  of  success. 

A  restricting  factor  in  a  differential  model  is  the  need  for  knowledge  of  boundary 
conditions.  This  seems  to  he  a  major  limitation  of  such  methods.  These  methods  apply 
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to  continuous  surface  patches  only  and  require  a  priori  knowledge  of  solution  values  at  some 
points  within  every  region.  This  means  that  we  must  find  regional  boundaries  —  perhaps 
ascertain  their  type  and  estimate  values  of  surface  orientation  at  some  points  within  each 
region  before  we  can  attempt  to  recover  shape.  Is  this,  in  effect,  placing  the  cart  before  the 
horse? 

Models  of  the  relationship  between  image  measurements  and  scene  variables  that  are 
formulated  as  low-order  differential  equations  offer  no  relief  from  the  necessity  of  knowing 
scene  parameters.  While  information  about  illumination  conditions  may  be  obtainable  from 
other  sources  within  the  image,  or  maybe  calculated  in  parallel  with  shape,  it  is  difficult  to 
envisage  a  situation  in  which  the  surface  albedo  could  be  calculated  before  the  surface  shape. 
Albedo  would  seem  less  constrained  than  shape.  The  author’s  higher-order  differential 
equations  show  that  derivatives  of  image  irradiance  can  be  used  to  remove  these  parameters. 

While  the  relaxation  schemes  used  to  solve  the  image  irradiance  equation  are  not  quite 
viable,  tbeir  drawbacks  may  be  attributed  to  the  weakness  of  the  surface  shape  constraint, 
namely  smoothness,  rather  than  an  inherent  deficiency  of  relaxation  as  a  technique.  For 
the  higher-order  surface-orientation-image-irradiance  equations,  insensitiveness  to  surface 
orientation  does  not  allow  assessment  of  the  strength  of  surface  continuity  (the  constraint 
used  in  the  attempts  to  solve  these  equations  by  relaxation).  The  results  reported  from 
these  relaxation  procedures  can  be  attributed  to  other  aspects  of  the  models  they  embody, 
rather  than  to  any  deficiency  of  the  relaxation  technique  itself.  Relaxation  seems  viable  as 
a  method  that  can  satisfy  global  constraints  without  being  dominated  by  numerical  error. 
However,  surface  shape  assumptions,  that  are  more  restrictive  than  those  used  in  the  work 
reviewed,  appear  necessary  if  information  is  to  be  propagated  effectively  over  reasonable 
image  distances.  Relaxation  schemes  that  implement  low-order  differential  models  seem 
practicable;  schemes  implementing  higher-order  differential  models  are  too  sensitive  to  noise. 

In  comparison  with  information  propagation  techniques,  local  computation  of  surface 
orientation,  as  reported  by  Pentland,  requires  strong  restrictions  on  surface  shape  —  and 
even  these  are  not  adequate  to  characterize  all  cases.  However,  local  computation,  par¬ 
ticularly  when  it  is  based  on  a  model  involving  derivatives  of  image  irradiance  only,  does 
provide  a  means  for  recovering  surface  orientation  without  any  knowledge  of  boundary  con¬ 
ditions,  without  a  priori  regional  segmentation  (it  may  even  help  in  this  endeavor),  and 
without  knowing  the  scene  parameters,  especially  albedo.  Unfortunately,  we  shall  not  get 
a  solution  to  surface  orientation  that  is  quantitively  correct  because  the  surface  restriction 
is  too  great.  Local  computation  offers  the  computational  features  we  want,  but  the  penalty 
to  be  paid  —  severe  surface  shape  restriction  —  is  far  too  great. 

What,  then,  seems  practical?  A  relaxation  scheme  that  is  more  constrained  by  surface 
type  than  those  that  have  been  examined?  A  scheme  that  implements  a  low-order  model  of 
information  propagation?  A  scheme  that  does  a  lot  of  purely  local  computation?  A  scheme 
that  can  use  boundary  conditions  wherever  they  are,  but  without  being  overly  dependent 
on  them?  Of  course,  all  this  is  one  conjecture.  There  may  well  be  a  group  of  models  that 
provide  purely  local  computation,  along  with  a  means  of  determining  when  each  model  is 
applicable.  Higher-order  differential  models,  however,  or  low-order  differential  models  that 
require  too  much  a  priori  scene  knowledge  do  not  appear  practicable.  For  any  realistic  model 
it  seems  inevitable  that  local  processing  must  play  an  important  role.  Consequently,  what 
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can  we  compute  locally  from  the  shading  data?  This  is  the  question  we  shall  now  address. 


4.  Analysis  of  Local  Computation 

The  relationship  between  surface  orientation  and  image  irradiance  for  a  uniformly 
diffuse  reflecting  surface  that  is  doubly  curved  is  given  by  the  surface-orientation-image- 
irradiance  equations  of  Section  2.3.  Parameter  counting  reveals  that  local  image  measure¬ 
ments  are  insufficient  to  specify  surface  orientation  for  the  general  case,  but  shape  constraints 
can  overcome  these  degrees  of  freedom.  Since  we  wish  to  calculate  surface  shape  locally,  we 
consider  the  case  in  which  we  can  assume  a  constant  curvature  over  the  small  surface  patch 
from  which  we  draw  information  for  the  local  calculation.  Of  course  the  curvature  varies 
with  direction;  we  only  assume  that  we  can  ignore  any  change  in  curvature  over  the  surface 
patch.  Of  course,  this  assumption  is  not  valid  in  general;  we  are  restricting  our  attention 
to  this  case  to  simplify  the  analysis.  If  we  cannot  determine  what  shape  information  is 
available  in  this  restricted  case,  we  are  not  likely  to  understand  the  general  case.  For  this 
case,  when  we  ignore  curvature  change,  =  lyy  —  txy  =  ttIxx  =  =  0,  and 

from  the  surface-orientation-image-irradiance  equations  we  derive  the  expressions 

Ixx  _  —  m^)  +  —  /^)  +  2lxmxlm 

hy  Ixlyil  —  m^)  +  mxTTiyil  —  P)  +  {IxtUy  +  lymx)lm 

lyy  _  ly^jl  ~~  TTp)  +  Tny^jl  -  P)  +  2lymylm 

Ixy  lxiy[^—rrP)  +  ^xmy[\  —  P)-ir{lxmy  +  lymx)lm  ' 

Notice  that  these  relationships  are  only  between  surface  shape  and  the  second  deriva¬ 
tives  of  the  image  irradiance.  It  is  the  assumption  of  constant  curvature,  not  the  more 
restrictive  sphericity  assumption  (used  by  Pentland  to  recover  surface  orientation  from  the 
second  derivatives  of  image  irradiance),  that  is  necessary  to  relate  shape  and  just  the  second 
derivatives  of  the  image  irradiance.  Image  measurements  are  generally  dependent  on  scene 
parameters  other  than  those  encoding  shape.  The  first  and  second  derivatives  of  image 
irradiance  depend  on  the  lighting  position  and  the  surface  albedo,  but  the  ratios  of  second 
derivatives  are  independent  of  all  scene  parameters  except  surface  shape. 

Can  we  use  the  above  expressions  to  calculate  surface  orientation?  We  have  previously 
[15]  pointed  to  the  insensitivity  of  surface-orientation-image-irradiance  equations  to  surface 
orientation.  The  above  expressions  are  also  insensitive  to  surface  orientation.  We  see  this 
in  the  following  considerations.  Algebraic  manipulation  of  the  above  expressions  yields 

Ixx  _  -H  mx^  —  (/jTTi  —  Intxf 

ty^  +  rny^  —  {lym  —  Imy)^ 

Suppose  that  over  an  image  patch  we  know  values  of  /  and  m  that  satisfy  the  above 
expression.  Consider  now  this  expression  for  ^  when 


t'  =  wyl  and  m'  =  W2Tn  , 
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at  each  point  of  the  image  patch.  Using  finite-difference  formulas  to  calculate  the  derivatives 
of  the  surface  normal,  we  obtain 


I'xx  _ 

I'yy  -h  m'j,-  -  -  /'m'yf 

_  -f  W2-mx^  -  -  Im^f 

+  ■W2^Tny^  —  Wi^Wz^lym  —  Iniyf 


Note  that,  as  the  magnitude  of  ivi  or  Wz  is  varied,  the  numerator  and  denominator  of  7^ 

‘  yu 

vary  in  like  manner;  both  either  increase  or  decrease;  tP  remains  approximately  equal  to 

7^  .  The  ratios  of  the  second  derivatives  of  image  irradiance  are  not  sensitive  to  surface 
orientation.  We  cannot  get  further  shape  information  from  other  image  measurements,  as 
the  first  and  second  derivatives  of  image  irradiance  are  dependent  on  the  surface  albedo  and 
the  lighting  conditions,  and  the  image  irradiance  is  dependent  on  surface  albedo,  lighting 
conditions,  and  the  level  of  constant  illumination  from  such  sources  as  atmospheric  scatter 
and  the  dark  current  of  the  sensor.  Surface  orientation  can  be  computed  locally  only  when 
very  restrictive  assumptions  about  surface  shape  are  made.  Without  such  restrictions  there 
is  not  enough  information  in  the  shading  to  decouple  surface  orientation  effects  from  those 
of  albedo  and  illumination. 

If  shading  is  insufficient  to  allow  surface  orientation  to  be  recovered,  what  then  does 
the  shading  specify?  Does  it  specify  curvature?  Can  we  compute  it  locally?  Consider  the 
above  expressions  for  and  Suppose  that  we  know  the  correct  values  for  I  and  m 
at  an  image  point  and  we  want  to  calculate  lx,ly,Tnx,  and  my.  If  lx,ly,fnx,  and  my  is  a 
solution,  then  so  is  wlxjwlyjwmx,  and  wmy,  where  w  is  any  constant.  Curvature  cannot  be 
computed  locally  (without  further  shape  assumptions).  The  ratios  of  second  derivatives  of 
image  irradiance  contain  shape  information,  yet  are  insensitive  to  surface  orientation  and 
do  not  allow  computation  of  the  curvature.  What  information  about  the  surface  do  they 
encode? 

To  answer  this  question,  we  first  rewrite  the  expressions  for  ^  and  ^  in  vector  dot 
product  form; 


Ixx  _  -  P  -  Tn^)\ 

\/l  -  -  m2)|.[^(/,m,  Vl  -  P  -m^)] 

^  m,  n/1  -  P  -  77^2)]  .^1  _  /2  _  ^2)] 

^^y  ~  v^l-/2-m2)].[^(/,  m,  m2)] 

Using  the  notation  N  =  (/,  m,  \/\  —  P  —  ),  for  the  unit  surface  normal,  we  obtain 

Ixx  _  N^-N, 

Ixu  N^.Ny  ’ 

-/yy 

l;  ^  N^.Ny  ^ 


where  N*  =  ^  and  Nj,  = 
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For  the  case  studied  —  when  curvature  changes  are  ignored  —  the  ratios  of  the  second 
derivatives  of  image  irradiance  measure  the  relative  squared  curvature  of  the  surface.  In 
other  words,  the  ratios  measure  the  relative  change  of  the  surface  normal  as  we  move  along 
orthogonal  image  directions.  However,  relative  curvature  calculated  locally  at  each  image 
point  constitutes  insufficient  information  to  allow  surface  shape  reconstruction  in  the  absence 
of  further  information  about  surface  parameters.  From  shading  information  alone  shape  is 
an  unattainable  goal. 

If  we  can  find  surface  shapes,  however,  for  which  knowledge  of  relative  curvature 
implies  stronger  information  about  the  surface,  e.g.,  surface  orientation  as  in  the  case  of  a 
sphere,  and  if  these  surface  shapes  are  reasonable  approximations  of  the  surfaces  found  in 
nature,  then  we  may  be  able  to  recover  stronger  shape  information  locally.  Locally  there  is 
not  enough  information  to  calculate  surface  shape  without  further  knowledge,  or  without 
additional  assumptions  about  surface  shape.  Pentland’s  work  shows  that  an  assumption 
of  sphericity  is  strong  enough  to  allow  surface  orientation  to  be  calculated  locally.  Is  this 
ability  to  calculate  surface  orientation  specifically  related  to  sphericity  —  or  is  it  a  feature 
that  is  generally  true  when  we  restrict  the  surface  shape  to  cases  in  which  the  number  of 
free  parameters  is  no  more  than  that  for  a  spherical  surface?  In  the  foregoing  discussion 
we  have  assumed  that  the  surface  is  doubly  curved.  We  shall  now  consider  the  images  of 
singularly  curved  surfaces. 

Just  as  we  did  for  doubly  curved  surfaces,  we  assume  that  the  derivatives  of  sur¬ 
face  curvature  can  be  ignored  when  we  consider  local  computation  of  surface  parameters. 
Differentiating  the  image  irradiance  equation,  we  obtain  the  same  expression  as  before  for 
the  doubly  curved  surface,  namely, 

Ian  _  +  rrix^  —  (/gTn  —  /mg)^ 

•^V!/  ly^  +  my^  —  [lyTTi  —  Irriyf' 


For  a  singularly  curved  surface  (IgT/iy  —  /j,m*  =  0)  when  surface  curvature  is  locally 
constant,  the  second  derivatives  of  image  irradiance  are  not  independent,  Ixxlyy  =  Ixy^- 
Consequently,  we  can  derive  only  one  expression  relating  shape  and  the  second  derivatives 
of  image  irradiance,  rather  than  the  two  expressions  we  derived  for  doubly  curved  surfaces. 
As  before,  it  follows  that 

Ixx  _  N,.N, 

^vv  NyNy 


At  first,  it  might  appear  that  there  is  more  shape  information  in  the  first  derivatives  of 
image  irradiance  for 

I^rTly  lyTTix  -  0  , 

hly-lylx=0  . 

But  this  is  not  the  case,  as  the  first  and  second  derivatives  of  image  irradiance  are  not 
independent.  For  singularly  curved  surfaces,  when  we  ignore  curvature  change,  ^  =  (^)^. 

For  the  singularly  curved  and  doubly  curved  surfaces  studied,  local  shading  specifies 
the  relative  curvature  of  the  surface  along  orthogonal  image  coordinates,  which  is  the  most 
we  can  hope  to  recover  by  local  computation.  In  general,  we  cannot  ignore  curvature  change 
over  a  patch.  In  this  case,  the  information  available  locally  in  the  image  combines  data  on 
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relative  curvature  and  curvature  change.  In  the  restricted  case  in  which  the  the  surface  is 
assumed  to  be  spherical  the  surface  orientation  can  be  calculated.  However,  this  appears  to 
be  a  very  special  situation  based  on  the  sphericity  assumption  rather  than  on  a  restriction 
in  the  number  of  parameters  needed  to  specify  the  surface.  Since  surfaces  in  general  are  not 
locally  spherical,  one  is  forced  to  conclude  that  shading  alone  cannot  enable  prediction  of 
surface  shape  by  purely  local  computation. 


5.  Conclusions 

The  recovery  of  a  scene’s  surface  shape  from  its  image  is  fundamental  to  the  vision 
process.  Our  purpose  in  processing  an  image  is  the  recovery  of  scene  properties,  not  those 
of  the  image  per  se.  In  remote  sensing  it  is  these  scene  properties  that  we  wish  to  measure, 
but,  to  extract  them,  we  have  to  understand  how  these  scene  properties  are  manifested  in 
the  image  data.  A  conceptual  model  of  the  relationship  between  scene  and  image  parameters 
is  provided  by  intrinsic  images.  Each  intrinsic  image  specifies,  for  each  point  in  the  image, 
the  value  of  one  of  the  scene  parameters  that  contribute  to  the  measured  image  intensity. 
Vision  models  try  to  recover  these  parameters  as  best  they  can,  whereupon  a  type  of 
relaxation  process  adjusts  their  values  so  that  they  constitute  a  consistent  interpretation 
of  the  scene’s  structure.  Which  parameters  are  specified  by  separate  intrinsic  images  and 
which  are  composite  is  unknown,  but  it  is  essential  that  they  be  estimable  without  the  need 
to  know  the  values  of  the  other  intrinsic  images.  Shape-from-shading  proposes  a  source  of 
information,  namely  shading,  from  which  shape  information  is  to  be  recovered  —  but  what 
shape  information  does  it  actually  encode? 

Local  shading  specifies  no  more  than  the  relative  curvature  of  the  scene’s  surface 
along  orthogonal  image  directions.  In  general,  even  the  recovery  of  relative  curvature  is 
complicated  by  change  in  the  curvature  of  the  surface.  However,  surface  shape  variables 
are  related  to  image  measurements  in  a  fashion  that  is  not  dependent  on  knowing  the  other 
scene  parameters.  Shading  provides  direct  shape  information,  but  this  is  not  enough  for 
reconstruction  of  the  surface  shape.  Further  relationships  between  shape  variables  and  image 
properties  must  be  established  before  shape  recovery  is  possible. 

The  various  approaches  reviewed  have  attempted  to  recover  surface  orientation  from 
shading.  To  do  so  they  have  added  extra  information,  such  as  known  boundary  conditions 
or  constraints  upon  surface  shape.  The  performance  of  these  various  models  allows  us  to 
draw  the  following  conclusions: 

•  Direct  integration  of  differential  models  of  scene  properties  requires  much  a  priori 
information  and  has  to  contend  with  major  computational  problems. 

•  Local  computation  must  play  a  major  role  in  the  recovery  of  scene  parameters,  but  the 
models  used  have  been  overly  restrictive  in  an  effort  to  recover  particular  information. 

•  A  relaxation  mechanism,  based  on  a  strong  low-order  differential  model,  seems  a  viable 
means  of  propagating  spatial  information  and  constraints. 

Shading  provides  a  basis  for  an  intrinsic  image,  specifying  relative  surface  curvature  and 
curvature  change,  but  this  intrinsic  image  alone  is  insufficient  for  surface  shape  recovery. 
Other  models  incorporating  other  image  measurements  are  needed  to  complement  shading. 
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Such  models  should  utilize  the  advantages  of  local  computation. 
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Appendix  A 


If  a  surface  is  twice  differentiable,  then 

ly(l  —  m^)  + TTiylm  =  mx{l  —  l^)  + Ixlrn  . 

We  call  this  the  surface  continuity  equation,  even  though  surface  continuity  is  less  demanding 
than  the  requirement  that  the  surface  be  twice  differentiable. 

Proof:  For  a  continuous  twice-differential  surface, 


Zxy  -  Zyx 


But  p  =  Zx  and  q  =  Zy,  so 
However, 


P  = 


9  = 


Py  9i 
-I 


Vl-P-  m2 

—m 


y/l-P- 


Hence, 


_  fy(l  —  m^)  +  ntylm 

Ps'  -  (1  _  _  „,2)3 

mx{l  —  P)  +  Ixlrn 


qx  =  - 


Then,  substituting  in  py  =  g*  yields 

ly{l  —  rrP)  +  mylm  =  mx{l  ~  P) -i- Ixlfn  . 


Appendix  B 

Developable  surfaces  are  characterized  by  the  differential  equation 

ixTTiy  lyTTlx  ””  0  • 

Proof:  With  the  e.xception  of  a  cylinder  whose  axis  is  parallel  to  the  z  axis,  the  differential 
equation  defining  all  developable  surfaces  is  [16] 

ZxxZyy  -  Zxy^  =  0  . 

As  the  surface  is  twice  differentiable,  then  Zxy  =  Zyx  so 

Zxx  ^yy  ^xy^yx  —  0 
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As  p  =  Zx  and  q  =  Zy  then 


Px^y  Py^x  —  0 


But 


-I 

v/1  -  P  -  m2 

—m 

\/\  —  P  —  rrfi 


} 


Hence, 


Ixil  —  rrP)  +  ruxlm 
/y(l  —  mP)  +  rriylm 

(1  —  /2  —  m2)i  ’ 
TTixi^  —  /^)  +  /*/m 
(l-/2-m2)§  ’ 

my(l  —  P)  +  lylm 

(l-/2_m2)5 


Substituting  in  pxqy  —  PyQx  =  0  gives 


Ixrriy  —  lyTIlx  =  0  . 


Appendix  C 

The  relationships  between  surface  orientation  and  image  irradiance  for  a  developable 
surface  are 

Ix^{rny{l  —  /^)  +  lylm)  =  —  m“)  +  mxlm)  , 

Tx^T^y  ~~  ^yZTXx  " —  0  , 

lxly~lylx=0  . 

Proof:  Differentiating  the  image  irradiance  equation,  I{x,  y)  =  R{l,m),  we  obtain 

Ix  Rllx  “b  Rfn^x  I 

Ty  ly  Rm  ZTty 


Now 

Ix\my[l~P)  +  lylm)=  Ri‘^lx{lxmy{l-P)  +  lxlylm) 

+i?,„^m2(m*my(l  —  P)  +  lymxlm) 
+2RiRmlx{mxTny{l  —  P)  +  lymxlm)  , 
lyH^xil  —  ttP)  +  mxlm)  —  Rt^ly(lxly(l  —  m^)  +  lymxlm) 

+Rm^my{lxmy[l  —  m^)  +mxmylm) 
+2RiRmly{lxmy{l  —  mP)  +  m^my/m)  . 
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But,  for  a  developable  surface  Ixtn^  —  lytrix,  (see  Appendix  B);  hence 


Ix^(my(l  —  /^)+  lylm)  —  —  l^)  +  l^lylm) 

+Iim^mximxTny(l  —  l^)  +  IxTUylm) 
+2RtRmlxifnxmy{l  —  l^)  + Ixmylm)  , 

—  m^)  +  nixlm)  =  i2j^/j,(/*/y(l  —  m^)  +  IxTUylm) 

+Rm^Tny{lymx(i.  —  m^)  +  Tn*mj,/m) 

—  TTl^)  +  TTlxTUylm)  . 

Therefore, 

/i‘(my(l  -  f^)  +  =(R[^lxly  +  Rm^m^my  +  2i?j/?m4»nj,)(m*(l  -  /^)  +  /a/m)  , 

—  m^)  +  mj/m)  ={Ri^lxly  +  Rm^ntxmy  +  2i?ji?m/y»«i)(^y(l  -  m^)  +  my/m)  . 

However,  the  surface  continuity  equation,  (see  Appendix  A),  is 

/y(l  —  m^)  +  my/m  =  ma(l  — /^)  + /a/m  . 

We  have  the  required  result,  i.e.,  that  the  relationship  between  surface  orientation  and  image 
irradiance  for  a  developable  surface  is 

Ix^imyil  —  /^)  +  /y/m)  =  +  rrixlm)  . 

In  terms  of  p  and  q,  the  equivalent  form  is 


Qy  ^y  Px  —  0 


In  terms  of  depth  z,  the  equivalent  form  is 


-l.Zx 


=  0  . 


Note  that,  in  addition, 

Ix^y  ^y^x  Rti^^x^y  ^y^x'}  j 
-f^/y  “  -/y/*  ~  Rfni^y^x  ^x^y) 

Hence,  for  a  developable  surface  /^my  —  lyirix  ==0,  we  obtain  the  required  results 

Tx^^y  JyTTlx  —  0  , 

^/y  —  lylx  =  0  • 
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